also to check: #6883

Up to standards ✅

_{TIP This summary will be updated as you push new changes. Give us feedback}

also to check: #6883

@BartChris would you mind reviewing this pull request, with that particular issue in mind, as you worked on the mass import quite a lot yourself?

When the uploaded csv file gets bigger i run into problems:

I uploaded a csv file with 453 rows:

When removing one of the red marked pseudo columns or when trying to switch the catalog the table is emptied:

When uploading a bigger csv file the "Add metadata" dialog is just an empty dialog:

vs

Edit: OK the fix for this button is the same:

   <p:commandButton id="addMetadataColumn"
                                         title="#{msgs['dataEditor.addMetadata.newMetadata']}"
                                         disabled="#{MassImportForm.records.size() eq 0}"
                                         process="@this"
                                         partialSubmit="true"
                                         style="margin: 3px"
                                         icon="fa fa-plus"
                                         styleClass="secondary"
                                         action="#{MassImportForm.addMetadataDialog.prepareMetadataTypes()}"
                                         update="addMetadataDialog"
                                         oncomplete="PF('addMetadataDialog').show();"/>

Maybe @henning-gerhardt can also check if his issue with the role settings is fixed, because the error always comes down to the number of elements edited.

I just noticed that the problems i encountered also appear in 3.9.0.

As we at SLUB did not use the mass import feature and all changes are only focused on mass import I don't think that this pull request is fixing #6805 in first place. The comments in #6805 are switching fast from the original issue to an other maybe related issue and so I don't think that the main issue is fixed with this changes here.

I tried it again and #6805 is already fixed by some changes between December 2025 and April 2026. So this pull request is not really related to the original issue of #6805 but maybe still to the comments in #6805.

I tried it again and #6805 is already fixed by some changes between December 2025 and April 2026. So this pull request is not really related to the original issue of #6805 but maybe still to the comments in #6805.

Yes, exactly. The linked issue was already fixed by #6810, I think, and the necessity to refactor the mass import form resulted from the changes in that pull request.

@BartChris thanks a lot for your review and suggestions, which fixed the problems you reported. (I commited them manually instead of applying the suggestions via GitHub because some indentations seemed to be lost in the suggestions).

I also increased the sizeLimit to a hopefully more reasonble value (and rebased against the current main branch).

Would you mind checking the pull request again?

I noticed that the MassImport button to trigger the import is never activated. (Maybe because of the changes i suggested)

I suppose we have to update it explicitely whenever we execute an action. I tested it with the catalogueSelect box and added massImportButtonForm to the list of to be updated components. Now the button is enabled after switching the catalog.

  <p:selectOneMenu id="catalogueSelect"
                                     disabled="#{not MassImportForm.firstColumnContainsRecordsIdentifier()}"
                                     required="#{not empty param['massImportButtonForm:importButton']}"
                                     converter="#{importConfigurationConverter}"
                                     value="#{MassImportForm.importConfigurationId}">
                        <f:selectItem itemValue="#{null}"
                                      itemLabel="-- #{msgs.selectCatalog} --"
                                      noSelectionOption="true"/>
                        <f:selectItems value="#{CreateProcessForm.catalogImportDialog.importConfigurations}"
                                       var="configuration"
                                       itemLabel="#{configuration.title}"
                                       itemValue="#{configuration}"/>
                        <p:ajax process="@this"
                                partialSubmit="true"
                                update="recordsForm massImportButtonForm"/>
                    </p:selectOneMenu>

This is not directly related to the PR and can maybe be fixed later, but i am wondering if we should spam the screen with tons of messages for every imported row:

Up to standards ✅

_{TIP This summary will be updated as you push new changes. Give us feedback}

This is not directly related to the PR and can maybe be fixed later, but i am wondering if we should spam the screen with tons of messages for every imported row:

Yes, that's suboptimal, and yes, we should address that in a separate pull request.

@BartChris I get a FacesException when trying to edit cell contents from slightly larger CSV files (about 100 rows), which I cannot explain, yet. Maybe it has something to do with lazy loading additional entries, but I don't now for sure.

@BartChris I get a FacesException when trying to edit cell contents from slightly larger CSV files (about 100 rows), which I cannot explain, yet. Maybe it has something to do with lazy loading additional entries, but I don't now for sure.

I do not get an error here (maybe you fixed that) but i also cannot edit and save the values (it keeps the old value). I would have to check if this ever worked on large tables.

I do not get an error here (maybe you fixed that) but i also cannot edit and save the values (it keeps the old value). I would have to check if this ever worked on large tables.

Yes, I was able to fix the problem with larger files by using the enctype="multipart/form-data" in the recordsForm, but editing CSV cells is currently not possible.

I tried updating the current row in

https://github.com/solth/kitodo-production/blob/c078685857381483bfff1233f0d19009c8d177da/Kitodo/src/main/webapp/WEB-INF/templates/includes/massImport/massImportTab.xhtml#L148

but that did not work, either.

I do not get an error here (maybe you fixed that) but i also cannot edit and save the values (it keeps the old value). I would have to check if this ever worked on large tables.

Yes, I was able to fix the problem with larger files by using the enctype="multipart/form-data" in the recordsForm, but editing CSV cells is currently not possible.

Did this work in 3.8? I would say this is a Mass Import and values should be provided by the CSV in general, so i do not see it as a critical problem, if values cannot be edited directly in the UI.

Did this work in 3.8? I would say this is a Mass Import and values should be provided by the CSV in general, so i do not see it as a critical problem, if values cannot be edited directly in the UI.

Yes, it did work in 3.8. And in 3.9 it works as well. In main, the mass import is pretty broken right now, though, to the point where it is not usable at all. The pull request does improve things and restores usabilty in general, but not being able to edit uploaded data (in contrast editing manually entered rows without a CSV upload seems to work fine) still is quite a regression when compared to previous releases, in my opinion.

@BartChris as a temporary step I deactivated the option to edit datatable cells after uploading a CSV file, so that this feature is not offered in a broken state.

I would suggest to go ahead and merge this pull request if you don't find any other problems with it, to at least restore the basic functionality of the mass import again. Once we get cell editing working again we can reactivate it afterwards in a separate pull request. What do you think?

Hmm, right now i got a problem when starting the import i have not seen before although i suppose this is not connected to the PR:

Hmm, right now i got a problem when starting the import i have not seen before although i suppose this is not connected to the PR:

We should probably make sure it isn't. Perhaps you could test with the same case in the main branch?

I just noticed that the "Mass import" button still isn't activated immediately after uploading a CSV files that already contains all data (thus metadata of type documentType in the first column) and doesn't require catalog queries.

I will check. Right now i suspect that this is again about external schema files i cannot resolve...

Up to standards ✅

_{TIP This summary will be updated as you push new changes. Give us feedback}

Ok, after commenting out

validationService.validateMappingFiles(importConfiguration.getMappingFiles());

and deactivating validation here

private String importProcessAndReturnParentID(String recordId, LinkedList<TempProcess> allProcesses,
                                                  ImportConfiguration importConfiguration, int projectID,
                                                  int templateID, boolean isParentInRecord, String parentIdMetadata,
                                                  boolean validateAgainstXmlSchema)

it works. So this is not directly connected and about some validation issue.

It should be possible to deactivate XML validation of external data for an import configuration in general, so you don't have to tamper with the code to achieve that:

Same error on main, so definitely not connected!

It should be possible to deactivate XML validation of external data for an import configuration in general, so you don't have to tamper with the code to achieve that:

In importProcessForMassImport the validation flag seems hardcoded and not influenced by the setting.

Edit:It seems to be additionally checked in convertDataRecordToInternal, sorry. But validateExternal is always true.

// validate external record against corresponding XML metadata schema(ta)
if (validateExternal && importConfiguration.getValidateExternalData()) {
           validationService.validateExternalRecord(xmlContent, importConfiguration, identifier);
}

But it seems like

// validate mapping files against xslt schema
validationService.validateMappingFiles(importConfiguration.getMappingFiles()); 

is always called. In my case i suppose the mapping file validation let the process fail.

In importProcessForMassImport the validation flag seems hardcoded and not influenced by the setting.

Good that we found this out so it can be fixed as well!

In importProcessForMassImport the validation flag seems hardcoded and not influenced by the setting.

Good that we found this out so it can be fixed as well!

Interestingly, the setting in the import configuration actually seemed to work for me.

When schema validation was activated in the configuration, I got the following result in mass import:

After deactivating the schema validation in the configuration, though, the import of the same IDs from the same catalog interface was succesful:

It seems the static parameter true in https://github.com/solth/kitodo-production/blob/e1fdd90482a14ce073dac77044db74c8a9e1884c/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java#L1577 does not actually control whether schema validation is applied during mass import or not 🤔

Maybe your import didn't really fail because of schema invalid metadata, but because the imported XML file or the import mapping XSLT did not contain valid XML to begin with?

Maybe your import didn't really fail because of schema invalid metadata, but because the imported XML file or the import mapping XSLT did not contain valid XML to begin with?

I have to check deeper, but my investigations right now seem to indicate that is not the validation of the catalog entries which fails for me, but the validation of the mapping files. And the latter might come down to the well known problem of external schema files, which are maybe blocked for me.

https://github.com/solth/kitodo-production/blob/e1fdd90482a14ce073dac77044db74c8a9e1884c/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java#L962

Edit: OK, i get the same error in "normal" catalog import... will make an issue next week.